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PREFACE 


This  Report  was  prepared  as  part  of  Rand's  DoD  Training  and  Man¬ 
power  Management  Program,  sponsored  by  the  Human  Resources  Rese;.rch 
Office  of  the  Defense  Advanced  Research  Projects  Agency  (ARPA) .  With 
manpower  issues  assuming  an  even  greater  importance  in  defense  planning 
and  budgeting,  it  is  the  purpose  of  this  research  program  to  develop 
broad  strategies  and  specific  solutions  for  dealing  with  present  and 
future  military  manpower  problems.  This  includes  the  development  of 
new  research  methodologies  for  examining  broad  classes  of  manpower  prob¬ 
lems,  as  well  as  specific  problem-oriented  research.  In  addition  to 
providing  analysis  of  current  and  future  manpower  issues,  it  is  hoped 
that  this  research  program  will  contribute  to  a  better  general  under¬ 
standing  of  the  manpower  problems  confronting  the  Department  of  Defense. 

Policy  analysts  are  relying  Increasingly  on  sample  survey  data 
collected  through  mail  questionnaires.  The  present  report  tries  to 
alert  mail  survey  users  to  the  common  sources  and  effects  of  non¬ 
response  and  to  inform  them  of  alternative  ways  of  dealing  with  non¬ 
response  bias.  It  does  this  by  examining  data  from  an  on-going  De¬ 
partment  of  Defense  survey  of  all  enlisted  men  who  were  separated 
from  active  service  after  30  June  1968. 
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SUMMARY 


Mail  surveys  have  several  noteworthy  advantages  (especially  low 
cost),  but  bias  due  to  nonresponse  poses  a  significant  threat  that 
users  of  mail  questionnaires  must  deal  with  either  by  maximizing  returns 
(and  thus  obviating  the  possibility  of  nonresponse  bias)  or  by  exer¬ 
cising  statistical  controls  to  attenuate  the  bias.  The  two  components 
of  nonresponse  bias  are:  (a)  nonoontacts  (e.g.,  invalid  addresses), 
and  (b)  nonreturns  (refusals) .  The  general  strategy  for  detecting  non¬ 
response  bias  is  to  compare  respondents  and  nonrespondents  on  a  variety 
of  dimensions  in  order  to  identify  the  direction  and  magnitude  of 
differences  between  the  two  groups,  and  this  is  typically  accomplished 
in  one  of  three  ways.  In  one  case,  a  subsample  of  people  who  did  not 
respond  to  the  mail  questionnaire  is  tracked  down  and  interviewed,  and 
then  their  responses  are  compared  with  the  responses  of  those  who 
answered  the  mail  questionnaire.  A  second  strategy  is  to  compare  the 
responses  that  came  in  before  and  after  follow-up  letters  went  out; 
the  assumption  here  is  that  "early"  and  "late"  respondents  are  ordered 
along  a  response  continuum,  and  that  nonrespondents  are  merely  "super 
late"  respondents.  And  finally,  nonresponse  bias  can  be  detected  by 
comparing  respondents  and  nonrespondents  on  dimensions  for  which  there 
are  data  available  for  all  potential  respondents. 

Using  the  third  strategy,  it  has  been  found  that  nonresponse  is 
related  to  a  number  of  factors  including:  (a)  ethnicity,  (b)  education, 
(c)  income,  (d)  age,  (e)  marital  status,  (f)  family  size,  (g)  geographic 
location,  and  (h)  interest  in  the  topic  of  the  survey.  Interest  in  the 
survey  Is  by  far  the  most  important  variable. 

The  replacement  of  missing  respondents  or  the  oversampling  of 
low  response  groups  are  examples  of  the  substitution  solution  to  non¬ 
response  bias,  but  these  procedures,  unfortunately,  simply  replace 
missing  respondents  with  people  like  the  ones  who  initially  responded. 
Thus,  substitution  strategies  increase  the  sample  size  which  has  been 
reduced  by  nonresponse,  but  they  do  not  correct  the  biased  composition 
of  the  sample. 


VII 

Another  approach  is  to  weight  subgroup  results  in  order  to 
attenuate  nonresponse  bias,  but  this  strategy  is  based  on  the  same 
tenuous  assumptions  as  the  direct  substitution  methods.  Weighting 
involves  three  steps:  (a)  selection  of  the  weighting  dimensions  — 
practical  considerations  usually  limit  the  number  of  variables  which 
can  be  used  to  five  or  less;  (b)  computation  of  the  weights  —  this 
involves  finding  the  reciprocals  of  the  response  proportions  for  each 
of  the  subgroups  which  are  created  by  cross-partitioning  the  population 
on  the  weighting  variables;  and  (c)  application  of  the  weights  to  the 
raw  data  in  order  to  find  the  adjusted  estimates. 
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INTRODUCTION 


In  1839  the  British  provoked  the  Opium  Wars  with  China  and  after 
three  years  of  fighting,  won  commercial  concessions  and  the  right  to 
govern  Hong  Kong.  During  the  same  period.  Upper  and  Lower  Canada  were 
joined  by  the  Act  of  Union ,  Dr.  Livingstone  began  his  explorations  of 
Africa,  Freidrich  Engels  was  scratching  out  Situation  of  the  Working 
Class  in  England ,  and  Ireland  was  swept  by  the  potato  blight  which  by 
1851  had  left  ver  one  million  people  dead  of  starvation  and  disease. 
Amidst  these  events,  the  first  scientific  report  of -a  mail  survey  went 
unnoticed  to  all  but  the  most  careful  readers  of  the  Royal  Statistical 
Society  Journal ,  and  although  the  respectability  of  this  method  has 
varied  over  the  years,  mail  questionnaires  still  offer  the  same  ad¬ 
vantages  that  led  the  English  clergy  to  conduct  a  mail  survey  of  parish 
resources  in  1838, 

This  brief  paper  describes  the  advantages  and  disadvantages  of  mail 
surveys  and  then  focuses  on  the  chief  problem  with  this  data  collection 
technique,  nonresponse  bias.  Specifically,  we  will  look  at  the  common 
characteristics  of  nonrespondents,  and  then  we  will  describe  alternative 
methods  of  detecting  and  correcting  nonresponse  bias. 

ADVANTAGES  AND  DISADVANTAGES  OF  MAIL  SURVEYS 

Briefly  summarized,  the  features  of  mail  surveys  which  make  them 
so  widely  used  are: 

o  Ability  to  cover  a  wide  geographical  area.  Interviewing 
over  a  wide  area  is  difficult  and  extremely  expensive. 

Getting  the  interviewer  to  the  respondent's  location  is 
a  major  cost  factor  in  personal  interviews,  especially 
where  two  or  three  call-backs  must  be  made  in  order  to 
make  contact.  In  effect,  the  mail  questionnaire  res¬ 
pondent  serves  as  his  own  interviewer. 
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0  Speed.  Mail  questionnaires  clearly  are  a  faster  data 
collection  method  than  personal  interviews,  and  if  the 
sample  size  is  very  large,  mail  surveys  may  be  faster 
than  telephone  interviews. 

o  Lack  of  Interviewer  Bias.  Much  research  has  demonstrated 
that  the  interviewer  is  not  a  neutral  element  in  face-to- 
face  interviews.  A  classic  case,  reported  by  Rice  in 
1929,  illustrates  well  the  problem  of  interviewer  bias. 

Two  trained  interviewers,  one  a  socialist  and  one  a  pro¬ 
hibitionist,  were  sent  to  interview  men  on  skid  row  in 
order  to  find  out  the  cause  of  their  destitution.  Men 
interviewed  by  the  socialist  interviewer  blamed  their 
plight  on  the  ills  of  the  economic  system,  and  those 
interviewed  by  the  prohibitionist  attributed  their  fate 
to  overindulgence  in  alcohol.  Racial  differences  are 
especially  important  sources  of  interviewer  bias. 
Respondents  tend  to  vary  their  answers  depending  upon 
the  race  of  the  interviewer.  Mail  questionnaires 
obviate  the  problem  of  interviewer  bias.* 

o  Anonymity.  Under  certain  circumstances,  self-administered 
questionnaires  may  produce  more  accurate  answers  than 
supposedly  higher-quality  face-to-face  interviews.  This 
is  especially  true  when  the  subject  of  the  questionnaire 
is  very  sensitive  or  controversial.  Knudsen,  Hallowell 
and  Irish  (.1967),  for  instance,  found  that  over  80 
percent  of  the  unwed  mothers  they  interviewed  agreed 
with  the  statement  "Premarital  sex  is  never  alright"  [sic] 
whereas  only  69  percent  of  another  group  of  unwed 
mothers  agreed  with  this  statement  when  it  was  asked 
in  a  sel '-administered  questionnaire. 


For  a  summary  of  the  literature  on  interviewer  bias,  see  Cannell 
and  Kahn  (1969) .  An  extensive  bibliography  on  interviewer  bias  is 
available  from  Dr.  Carol  Weiss,  Bureau  of  Applied  Social  Research, 
Columbia  University,  New  York,  10027. 
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o  Ability  To  Consult  Records.  The  mail  questionnaire  does  not 
have  to  be  completed  within  a  fixed  period  of  time  as  does 
an  interview,  and  this  means  that  respondents  can  take 
time  to  consult  their  records  or  friends  before  answering 
a  complex  question.  This  is  especially  useful  in  gathering 
data  about  income  and  expenses,  health  histories,  and  similar 
factual  information. 

o  Low  Cost.  Mail  questionnaires  are  relatively  inexpensive, 
and  this  is  the  main  reason  they  are  so  widely  used.  The 
average  cost  of  a  mail  questionnaire  is  about  $1.50  to 
$3.50,  depending  upon  the  number  of  follow-ups,  the 
initial  response  rate  and  the  overall  quality  of  the 
materials  used.  Compare  this  with  the  typical  cost  of 
$35.00  to  $100.00  for  a  personal  interview  (again  de¬ 
pending  upon  the  number  of  call  backs,  the  length  of 
the  questionnaire  and  the  distance  between  respondents) , 
and  the  cost  of  $3.50  to  $7.50  for  a  telephone  interview. 

The  chief  disadvantage  of  mail  questionnaires  is  the  problem  of 
bias  due  to  nonresponse.  If  the  survey  does  not  achieve  a  high  com¬ 
pletion  rate,  the  results  may  not  represent  the  entire  target  population 
well.  If  only  fifty  percent  of  the  potential  respondents  reply,  the 
sample  data  probably  generalize  well  to  1  out  of  every  2  people  .n  the 
population,  but  which  one  out  of  two? 

The  problem  of  self-selection  or  self-assignment  to  conditions  in 
true  experiments  is  well  known,  and  few  researchers  would  accept  as 
valid  an  "experiment"  that  permitted  either  self-assignment  to  con¬ 
ditions  or  a  high  attrition  rate  from  conditions.  But  we  are  more 
generous  when  it  comes  to  accepting  self-selection  artifacts  in  surveys 
or  field  experiments  because  we  realize  the  tremendous  difficulties  in 
achieving  a  truly  random  sample  of  noncaptive  populations.  Nonethe¬ 
less,  the  self-selection  artifacts  are  there;  and  simply  admitting 
that  these  problems  are  difficult  to  eliminate  does  not  reduce  the 
threat  they  pose. 
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The  problem  of  nonresponse  bias  is  well  illustrated  by  an  old 
study  of  commercial  peach  production  which  was  conducted  in  1946  by 
Finkler  (1950).  Three  mailings  were  used  to  survey  3116  growers  in 
North  Carolina,  and  one  of  the  survey  questions  asked  how  many  trees 
the  grower  owned;  fortunately,  the  actual  number  of  trees  was  already 
known.  Finkler's  results,  as  summarized  by  Cochran  (1963,  p.  356), 
are  shown  in  Table  1.  The  presence  and  direction  of  nonresponse  bias 
are  obvious  here:  Growers  with  large  holdings  were  more  likely  to 
respond  than  owners  of  small  orchards.  Population  estimates  based  on 
the  sample  returns  imprjved  (i.e.,  came  closer  to  the  true  population 
mean)  as  the  number  of  returns  increased,  but  even  with  a  return  of 
41  percent  after  three  mailings,  the  sample  estimates  greatly  over¬ 
estimated  the  number  of  trees  that  the  average  grower  owned. 


Table  1 

RESPONSES  TO  THREE  REQUESTS  IN  A  MAILED  INQUIRY 
(Cochran,  1963,  p.  356) 


Number 

of 

Growers 

Percent 

of 

Population 

Average  Number 
of  Fruit  Trees 
per  Grower 

Response  !;_•  first  mailing 

300 

10 

456 

Response  to  second  mailing 

543 

17 

382 

Response  to  third  mailing 

434 

14 

340 

Nonrespondents  after  3 
mailings 

1,839 

59 

290 

Total  Population 

3,116 

100 

329 

COMPONENTS  OF  NONRESPONSE 

By  definition,  nonresponse  bias  occurs  whenever  a  nonrandom 
sample  of  potential  respondents  does  not  complete  and  return  the  mail 
questionnaire.  In  raai  .  surveys,  the  two  major  sources  of  dross,  that 
is  wastage  or  loss,  are  invalid  addresses  (noncontacts)  and  outright 
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refusals  (nonreturns);  and  in  computing  response  rates,  both  of  these 
sources  of  dross  must  be  included  in  the  denominator  along  with  the 
number  of  completed  returns.  The  reason  we  include  invalid  addresses 
in  figuring  response  rates  should  be  clear.  If  we  cannot  get  the 
questionnaire  into  a  potential  respondent's  hands,  he  cannot  complete 
the  questionnaire  and  return  it.  If  he  does  not  return  it,  he  is  not 
included  in  the  survey  results,  and  his  particular  responses  are 
therefore  ignored  in  making  population  estimates.  It  is  obvious  that 
noncontacts  are  just  as  dangerous  in  mail  questionnaires  as  outright 
refusals  (nonreturns)  unless  the  noneontaets  occur  randomly  with  respect 
to  the  dependent  varieties  of  interest. 

DEALING  WITH  NONRESPONSE  BIAS 

There  are  basically  two  ways  of  dealing  with  the  potential  problem 
of  nonresponse  bias  in  mail  surveys.  The  first  approach  is  to  do 
everything  possible  to  maximize  the  return  rate  and  thus  to  obviate 
the  whole  problem  of  nonresponse  bias.  Years  of  experience  have  pro- 
duced  a  large  bag  of  tricks  for  maximizing  response  rates.  Among  the 
important  factors  are:  incentives,  follow-ups,  forewarnings  (preview 
notices),  questionnaire  characteristics  (length,  color,  subject  matter), 
cover  letter  characteristics  (length,  type  of  appeal,  format,  ethnicity 
and  status  of  signer),  deadlines,  time  of  posting,  type  and  amount  of 
outgoing  and  return  postage,  and  respondent  anonymity. 

A  second  way  of  dealing  with  nonresponse  bias  is  to  examine  the 
returns  in  order  to  detect  potential  bias  (hoping,  of  course,  that 
none  is  found)  and  estimate  its  direction  and  magnitude.  In  the  event 
significant  bias  is  detected,  the  only  strategy  left  is  to  fall  back 
on  statistical  controls  to  offset  the  bias. 

DETECTING  NONRESPONSE  BIAS 

General  lore  among  users  of  mail  surveys  is  that  a  return  of  80 
percent  is  sufficient  tc  eliminate  major  nonresponse  bias;  at  that 
point,  getting  the  entire  sample  population  to  respond  would  not 
change  the  sample  estimates  significantly. 

Erdos  (1970)  provides  a  summary  of  these  techniques  for  maximizing 
response  rates. 
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When  returns  fall  below  80  percent  (but  above  the  obviously  low 
minimum  of  50  percent),  efforts  are  made  to  detect  the  possible  bias, 
and  there  are  basically  three  ways  in  which  this  can  be  done.  They 
are,  in  the  order  of  their  elegance: 

a)  Draw  a  subsample  of  the  nonrespondents  and  make  every 
effort  to  complete  telephone  or  personal  interviews 
with  these  people,  and  then  compare  their  responses  with 
those  of  regular  respondents  to  sfe  if  there  are  any 
obvious  differences.  This,  and  all  of  the  other  attacks 
on  response  bias,  suffer  from  a  particular  version  of 
the  matching  fallacy.  We  can  never  be  sure  that  we  have 
tapped  all  of  the  relevant  dimensions  in  our  matching 

or  comparisons  (Campbell  and  Stanley,  1966;  Gross, 

Collins,  and  Bryan,  1972,  p.  16). 

b)  Compare  the  responds  from  "early  respondents"  (prefollow-up 
letters)  with  those  collected  from  late  respondents"  (post- 
follow-up  letters)  to  see  if  there  are  any  obvious  trends 
operating.  E.g.,  are  "late"  respondents  younger  or  less 
educate'*  than  "early"  respondents?  This  of  course  assumes 
that  trends  are  operating  and  that  nonrespondents  are 
simply  super-late  respondents;  both  assumptions  are 
questionable. 

c)  Compare  respondents  and  nonrespondents  on  dimensions  for 
which  complete  data  are  available  for  all  elements  of  the 
sampling  frame.  For  example,  we  can  usually  infer  the 
potential  respondent's  sex,  so  response  rates  for  males 
and  females  can  be  compared.* 

A  number  of  studies  have  used  these  three  methods,  particularly 
the  latter,  to  estimate  the  direction  of  nonresponse  bias.  Some  of 

* 

The  availability  of  1970  Census  data  (Fifth  Count  Summary  Tapes) 
coded  to  ZIP  Code  zones  presents  a  new  source  of  aggregate  data  which 
can  be  used  to  estimate  some  demographic  characteristics  of  respondents 
and  nonrespondents,  or  at  least  their  neighborhoods 
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Che  dimensions  on  which  respondents  and  nonrespondents  have  been 

* 

compared  include  the  following: 

o  Education.  In  general,  education  is  positively  corre¬ 
lated  with  the  return  rate  (Baur,  1947;  Britton  and 

Britton,  1951;  Gaudet  and  Wilson,  1940;  Benson,  Booman 

and  Clark,  1951;  Lundberg  and  Larsen,  1949;  Sobol, 

1959;  Rooins,  1963;  Suchman  and  McCandless,  1940),  but 
in  some  cases,  highly  educated  people  (e.g.,  physicians 
and  lawyers)  are  hard  to  contact  in  mail  surveys. 

o  Income .  Income  and  response  rates  appear  to  be  curvi- 

linearly  related;  low  income  persons  are  especially 
difficult  to  contact,  and  even  among  those  contacted 
there  is  typically  a  very  low  response  rate.  At  the  other 
end  of  the  income  continuum.,  rich  people  are  also  diffi¬ 
cult  to  reach  because  they  live  ensconced  behind  a 
screen  of  secretaries  and  aides.  (Parten,  1950; 

Harding,  1944;  Ferber,  1956;  Ston borough,  1942; 

Vincent,  1964). 

o  Age .  Young  people  are  difficult  to  contact  because 
of  their  frequent  shifts  in  residence  and  their  active 
life  style.  Elderly  people  are  also  difficult  to  survey, 
because  they  tend  to  have  a  low  response  rate  to  mail 
questionnaires,  even  though  they  can  be  located  fairly 
easily.  This  probably  reflects  a  general  narrowing  of 
interests  in  matters  beyond  their  immediate  day-to-day 
environment . 


For  an  excellent  summary  of  the  differences  between  respondents 
and  nonrespondent j  in  mail  surveys,  see  Ursula  Dibble  and  Charles 
Kadushin,  "Factors  Which  Contribute  to  Response  Bias  in  Surveys  and 
Panels,"  New  York:  Market  Research  Corporation,  1967,  (Mimeo) . 

For  a  more  general  bibliography  on  mail  questionnaires,  see  R.  G. 
Bridge,  Mail  Questionnaires ,  Los  Angeles,  UCLA  Survey  Research 
Center,  Technical  Paper  No.  7002,  1970. 
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o  Ethnicity.  At  least  one  study  (Davis,  1964)  found  that 
blacks  had  a  lower  response  rate  than  whites  in  a  panel 
mail  survey.  But  simple  generalizations  must  be  avoided 
here,  for  at  this  point  in  American  history,  ethnicity  is 
highly  correlated  with  income;  and  income,  not  ethnicity, 
may  be  the  variable  which  actually  affects  response  be¬ 
havior. 

o  Rural  vs  Urban  Location.  People  living  in  rural  areas 
generally  show  a  higher  return  rate  than  city  dwellers 
(e.g.,  Reuss,  1943).  This  parallels  evidence  that 
personal  interviews  are  harder  to  complete  in  cities 
than  in  rural  areas  because  city  dwellers  spend  less 
time  in  their  homes  and  are  therefore  more  difficult  to 
contact.  On  the  other  hand,  the  growing  number  of 
"security  buildings"  that  exclude  outsiders  —  including 
interviewers  —  is  making  the  mail  questionnaire 
relatively  more  attractive  for  surveys  of  urban  popu¬ 
lations.  Mail  can  get  in  where  an  interviewer  cannot. 

o  Family  Size.  Young  children  keep  parents  close  to  home, 
and  therefore  it  should  not  be  surprising  to  find  that 
people  with  young  children  are  easier  to  contact  than 
those  with  older  children  (Hilgard  and  Payne,  1944)  or 
couples  with  no  children  (Ferber,  1956;  Lowenstein, 

Colombotos  and  Elinson,  1962). 

o  Marital  Status.  Unmarried  people  generally  show  a  lower 
response  rate  than  married  persons,  but  this  of  course 
is  because  marriage  is  confounded  with  a  host  of  other 
factors  including  age  and  family  size. 

Other  variables  on  which  respondents  and  nonrespondents  have  been 
compared  include:  sex,  letter  writing  activity,  socioeconomic  status, 
size  of  company,  and  interest  in  survey  topic.  The  single  most 
important  factor  is  the  topic  of  the  survey;  if  the  topic  interests 
the  respondent,  and  he  has  something  positive  to  report  about  himself, 
he  is  very  likely  to  complete  the  questionnaire.  Ball  (1930),  for 
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example,  found  chat  postcard  questionnaires  sent  to  families  of 
juveniles  on  probation  were  most  often  returned  by  parents  who  had 
something  positive  to  report  about  their  child  and  least  often 
returned  by  those  parents  who  were  having  trouble  with  their  children. 
Edgerton,  Britt,  and  Norman  (1947)  found  that  winners  in  a  science  fair 
contest  were  most  likely  to  respond  to  a  follow-up  survey,  honorable- 
mention  recipients  were  next  most  likely  to  respond,  and  "also  rans" 
were  least  likely  to  respond.  In  an  earlier  study.  Pace  (1939)  found 
that  alumni  who  had  graduated  from  a  university  were  more  likely  than 
dropouts  to  respond  to  a  questionnaire  from  the  university,  and  further¬ 
more  the  likelihood  of  responding  was  directly  related  to  the  nuc !  r 
of  quarters  the  individual  had  spent  at  the  school.  Similarly, 
Shuttleworth  (1941)  found  that  successful  alumni  were  more  likely  than 
unsuccessful  graduates  to  respond  to  a  university  survey.  And,  finally, 
Stanton  (1939)  found  that  teachers  who  used  radios  in  the  classroom 
were  most  likely  to  respond  to  a  survey  about  the  educational  use  of 
radio. 

The  list  of  examples  is  endless,  but  the  important  thing  to 
remember  is  that  the  dross  in  most  mail  surveys  is  not  random. 
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ILLUSTRATION:  A  POST-SERVICE  SURVEY  OF  ENLISTED  MEN 


This  section  provides  an  illustration  of  nonresponse  bias  in  a 
large  mail  survey.  The  data  were  collected  in  an  on-going  Departuent 
of  Defense  survey  of  all  enlisted  men  who  were  separated  from  active 
duty  after  30  June  1968.  These  data  were  gathered  in  order  "to  evalu¬ 
ate  the  effectiveness  of  such  programs  as  Project  One  Hundred  Thousand 
.  .  .  and  to  determine  the  extent  of  utilization  in  the  civilian 
economy  of  skills  developed  during  military  service  (DoD  Instruction 
1125.5,  1968,  page  E-2)."  The  results  of  these  surveys  are  especially 
important  now  that  the  American  military  is  constrained  to  operating 
an  all-volunteer  force  (AVF);  military  manpower  decisions  have  taken 
on  a  new  urgency  and  margins  for  error  have  been  reduced  greatly  in 
this  no-draft  environment.  The  post-service  surveys  are  an  important 
source  of  information  about  the  effects,  if  any,  of  the  military's 
large  scale  compensatory  education  programs  (e.g.,  Project  100,000). 

The  post-service  questionnaire  is  sent  to  each  enlisted  man 
approximately  ten  months  after  he  separates  from  active  duty,  and  if 
no  reply  is  received,  at  least  one  follow-up  letter  is  sent  by 
registered  mail.  The  completed  mail  questionnaire  yields  nine  ele¬ 
ments  of  information  including:  (1)  current  reserve  status, 

(2)  marital  status,  (3)  number  of  dependents,  (4)  highest  year  of 
education  completed,  (5)  employment  status,  (6)  hours  of  work  per 
week,  (7)  civilian  occupation  (DOT  classification),  (8)  income,  and 
(9)  educational  or  training  program  status. 

The  questionnaire  responses  are  merged  with  extant  records  from 
an  End  of  Service  file  in  order  to  create  a  Post-service  File.  The 
End  of  Service  File  provides  information  about  each  potential  re¬ 
spondent,  and  this  is  particularly  important  to  methodological 
researchers  because  it  means  that  the  attributes  of  respondents  and 
nonrespondents  can  be  compared.  For  each  potential  respondent,  the 


For  a  brief  description  of  Project  100,000,  the  program  under 
which  "unqualified"  recruits  were  taken  into  the  Army  and  given 
special  training,  see  Wool  and  Flyer  (1969). 
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End  of  Service  file  provides  23  pieces  of  Information  Including: 

(1)  service,  (2)  date  of  birth,  (3)  race,  (4)  whether  he  was  inducted 
or  enlisted,  (5)  home  and  mailing  addresses,  (6)  AFQT  percentile 
score,  a  measure  of  aptitude,  (7)  primary  and  secondary  military 
specialties,  (8)  pay  grade,  (9)  date  of  separation,  (10)  marital 
status,  (11)  number  of  dependents,  (12)  educational  background, 

(13)  total  months  in  the  military,  (14)  whether  or  not  he  participated 
in  special  military  programs,  and  (15)  whether  or  not  he  saw  duty  in 
Southeast  Asia. 

The  illustrations  used  in  this  paper  are  based  on  data  for 
163,666  of  the  176,018  Army,  Air  Force  and  Navy  enlisted  men  who  were 
separated  from  active  duty  during  Fiscal  Year  1971,  and  who  served  in 
selected  military  occupations.*  The  12,352  cases  were  omitted  for 
the  following  reasons:  Some  3099  men  were  not  required  to  participate 
in  the  survey,  AFQT  scores  were  not  available  for  8,568  men,  and  the 
educational  backgrounds  of  446  men  were  unknown.  For  convenience, 

239  men  who  were  not  classified  as  either  Caucasian  or  Negro  were 
omitted.  Using  these  data,  we  will  first  look  at  the  bias  introduced 
by  invalid  addresses,  and  then  we  will  examine  the  combined  effects 
of  nonrontacts  and  nonreturns. 

INVALID  ADDRESSES  (NONCONTACTS) 

The  FY71  post-service  survey  proves  to  be  no  exception  to  the  gen¬ 
eral  rule  that  dross  occurs  nonrandomly.  Here,  invalid  addresses  were 
related  to  the  potential  respondent's  service  affiliation,  race,  AFQT 

class,  education  and  enrollment  in  special  programs  (Project  100,000  or 
** 

Project  Transition  ).  Table  2  shows  the  percentage  of  invalid  addresses 
according  to  those  categories.  Briefly,  these  data  suggest  that: 


The  sample  includes  all  enlisted  men  in  Military  Occupational 
Specialty  classifications  1  (electronics),  2  (comnunications/intelli- 
gence)  and  3  (medical),  plus  those  in  Primary  MOS  categories  11  (infantry) 
and  68B  (aircraft  engine  repair)  and  all  Navy  aircraft  mechanics  (AD,  ADJ), 
and  Air  Force  jet  engine  mechanics  (432x0),  and  all  with  classification 
number  400,  610,  661,  702,  712,  721,  780,  and  800. 

A* 

Project  Transition,  a  voluntary  program  that  began  in  1067,  makes 
job  counciling,  entry  level  skill  training,  and  placement  services  avail¬ 
able  to  enlisted  men  anticipating  discharge. 
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o  The  Navy  evidently  had  the  best  set  of  records,  for  they 
turned  up  only  about  one  percent  invalid  addresses  in 
their  mailing.  In  contrast,  the  Army  mailing  lists  con¬ 
tained  almost  six  percent  invalid  addresses.  (No  data 
are  available  for  the  Air  Force).  But  from  these 
differences  in  the  percentage  of  invalid  addresses 
we  should  not  infer  that  the  Navy  does  a  better  job 
of  record  keeping  than  the  Army.  While  this  may  be 
true,  there  is  another  reason  that  the  records  for 
these  two  services  should  differ  in  accuracy;  the 
populations  in  these  two  services  are  markedly 
different,  and  these  differences  are  related  to 
stability  of  residence. 

o  About  five  percent  of  the  enlisted  men  could  not  be 

contacted  because  of  invalid  addresses,  and  this  dross 
is  strongly  related  to  race.  Four  percent  of  the  white 
enlisted  men  could  not  be  reached,  but  for  black  enlisted 
men  the  rate  was  11-1/2  percent.  One  out  of  nine  black 
respondents  never  even  received  the  questionnaire,  so 
clearly  the  survey  results  will  underrepresent  this  group 
(or  overrepresent  white  respondents) ;  but  the  issue  is 
not  simply  one  of  racial  differences  for  it  is  undoubtedly 
true  that  the  11-1/2  percent  of  the  black  population  which 
did  not  receive  a  questionnaire  is  not  a  random  sample 
of  all  black  respondents. 

o  Invalid  addresses  were  related  to  education.  Almost  ten 
percent  of  the  potential  respondents  with  less  than  a 
high  school  education  could  not  be  contacted;  the  non- 
contact  rate  dropped  to  4-1/2  percent  for  high  school 
graduates,  and  for  those  who  had  one  to  three  years  of 
college,  the  noncontact  rate  was  three  percent.  For 
college  graduates,  the  figure  was  only  two  percent. 

o  The  dross  due  to  invalid  addresses  was  a  monotonically 
decreasing  function  of  AFQT  scores.  Only  two  percent 
of  the  highest  AFQT  scorers  (Class  I,  centiles  93  -  99) 
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were  lost  due  to  invalid  addresses;  in  contrast,  ten 
percent  of  the  lowest  AFQT  scorers  (centiles  0  -  10) 
were  lost. 

o  People  who  participated  in  Project  100,000  were  harder 
to  locate  than  people  who  participated  in  Project 
Transition  (or  both  programs).  The  observed  differences 
probably  reflect  differences  in  the  populations  served 
by  the  two  programs,  however  the  important  point  is 
that  one  out  of  every  ten  participants  in  the  military's 
chief  compensatory  training  program,  Project  100,000, 
could  not  be  located.  Thus,  we  have  a  built-in  under¬ 
representation  of  the  participants  in  Project  100,000. 

We  may  conclude  that  noncontacts  (invalid  addresses)  did  not 
occur  randomly  in  this  mail  survey.  This  is  a  significant  source  of 
nonresponse  bias,  but  nonreturns,  i.e.,  outright  refusals,  represent 
an  even  larger  component  of  nonresponse  bias,  and  in  the  next  sub¬ 
section  we  will  consider  the  combined  impact  of  noncontacts  and  non¬ 
returns  . 

NONRESPONSE  (NONRETURNS) 

Altogether,  163,666  questionnaires  were  mailed  out  to  this  sample 
of  enlisted  men,  and  109,875  were  returned  either  partially  or  totally 
completed.  This  represents  an  overall  response  rate  of  67  percent, 
but  this  overall  figure  masks  some  important  subgroup  differences. 
Table  3  shows  the  percent  of  people  who  responded,  broken  down  by 
Service,  race,  AFQT  class,  education,  marital  status,  and  source  of 
accession  (enlistment/inducation) .  We  find  that  the  nonreturns  were 
anything  but  random. 

o  The  Air  Force  turned  in  the  best  response  rate,  75 
percent;  and  the  Army,  with  a  much  larger  population 
(n  =  131,108)  was  close  with  a  return  of  72  percent. 

The  Navy,  with  the  smallest  population  of  the  three 
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Table  2 

PERCENT  AVD  NUMBER  OF  RESPONDENTS  NOT  CONTACTED  BECAUSE  OF  INVALID  ADDRESSES 
PRESENTED  ACCORD :NC  TO  SERVICE,  AFQT  CLASS,  RACE,  EDUCATION, 

AND  PARTICIPATION  IN  SPECIAL  PROGRAMS 


Percent 

Number 

of 

of 

Addresses 

Addresses 

Total 

Category 

Invalid3 

Invalid 

(N) 

Service 

Armv 

5.6 

7,366 

131,108 

Navy 

1.2 

280 

23,171 

5.0 

7,646 

154,279 

Race 

Caucasian 

4.1 

5,511 

135,704 

Negro 

11.5 

2,135 

18,575 

5.0 

7,646 

154,279 

AFQT 

Class 

Class  I  (93-99  centile) 

2.3 

272 

11,857 

Class  II  (65-92  centile) 

3.0 

1,461 

49,216 

Class  III  (31-64  centile) 

5.1 

2,808 

55,452 

Class  IV  (10-30  centile) 

8.2 

3,012 

-  .  01 

Class  V  (below  10  centile) 

9.3 

93 

953 

5.0 

7,646 

154,279 

Less  than  8  yr 

8.3 

100 

1,205 

8  yr 

10.0 

352 

3,530 

9  yr 

9.5 

497 

5,227 

10  yr 

10.1 

760 

7,519 

11  yr 

9.5 

816 

8,579 

Education 

12  yr 

4.5 

4,149 

92,237 

1  yr  college 

3.1 

442 

14.081 

2  yr  college 

2.8 

267 

9,570 

3  yr  college 

2.4 

104 

4,334 

4  yr  college 

2.0 

146 

7,341 

M.  A. 

1.5 

5 

328 

Ph.D. 

2.4 

8 

328 

5.0 

7,646 

154,279 

Special 

Programs 

Project  100,000 

Project  Transition 

10.4 

3.4 

607 

517 

5,823 

15,124 

100K  and  Transition 

2.5 

27 

1,080 

^This  table  excludes  Air  Force  enlisted  men,  for  whom  no  data  are  available  on 
invalid  addresses. 
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services,  turned  In  the  lowest  response  rate,  a 
miserable  38  percent.* 

Returns  were  related  to  race.  For  whites  the  rate 
was  69  percent,  while  for  black  enlisted  men  the 
rate  was  51  percent.  Black  enlisted  men  were  under¬ 
represented  for  two  reasons:  first  they  were  con¬ 
tacted  less  often  than  whites,  and  second  they  were 
less  likely  to  respond  than  whites.  The  result  is 
that  the  population  estimates  based  on  the  survey 
responses  will  be  very  much  biased  against  black 
respondents . 

AFQT  scores  were  also  related  to  response  rates,  but 
the  relationship  was  weak.  Sixty-nine  percent  of  the 
enlisted  men  with  AFQT  scores  in  Classes  I,  II,  and 
III  (centiles  31  through  99)  responded  whereas  only 
61  percent  of  those  with  scores  in  Classes  IV  and  V 
responded. 

Married  men  were  more  likely  than  single  men  to  respond; 
the  response  rate  was  77  percent  for  married  men  versus 

hctc 

57  percent  for  unmarried  men. 

Upon  first  examination,  it  appears  that  draftees  were 
more  likely  than  enlistees  to  respond  to  the  question¬ 
naire;  the  return  rates  for  inductees  and  enlistees 
were  75  percent  and  58  percent,  respectively.  But 
this  apparent  difference  is  an  artifact  of  the  low 


The  Navy's  relatively  low  response  rate  is  somewhat  surprising, 
because  it  would  appear  that  the  Navy  population  should  have  a 
generally  higher  propensity  to  respond.  Relative  to  the  Army  and 
Air  Force  populations,  the  Navy  population  has  a  higher  average  AFQT 
score,  more  education,  a  larger  percent  Caucasian,  and  more  people 
at  higher  pay  grades.  All  of  these  variables — intelligence,  education, 
ethnicity  and  income — are  usually  positively  correlated  with  response 
behavior.  Probably  the  best  explanation  for  the  differences  between 
the  Services'  response  rates  is  that  they  each  used  different  survey 
procedures  (e.g.,  number  or  type  of  follow-ups),  but  this  is  only 
speculation;  unfortunately,  the  precise  survey  procedures  which  were 
used  were  not  well  documented. 

**Divorced  and  never-married  men  were  classified  as  single. 
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response  rate  for  Navy  personnel.  The  Inductees  served  in 
the  Army;  in  fact,  draftees  account  for  66  percent  of  all 
the  Army  men  in  the  sample.  Their  return  rate  of  75  percent 
is  very  close  to  the  all-Army  return  of  72  percent,  and 
therefore  we  may  conclude  that  there  was  no  important  differ¬ 
ence  between  the  response  rates  of  Army  enlistees  and  induct¬ 
ees  in  this  particular  sample.  A  potential  respondent's 
educational  background  was  also  a  good  predictor  of  his  like¬ 
lihood  of  responding.  Sixty  percent  of  those  with  less  than 
12  years  of  school  responded,  whereas  the  rate  for  those 
with  a  high  school  diploma  was  67  percent,  and  the  rate  for 
those  who  had  one  to  three  years  of  college  was  71  percent. 

The  highest  response  rate,  83  percent,  came  from  college 
graduates . 

o  Participants  in  Project  100,000  were  less  likely  to  respond 
than  nonparticipants,  but  those  men  who  participated  in 
Project  Transition  were  more  likely  than  nonparticipants  to 
respond.  Again,  these  differences  reflect  differences  between 
the  populations  which  are  served  by  Project  100,000  and 
Project  Transition. 

CONCLUSIONS 

The  dross  in  the  post-service  file,  at  least  for  the  FY71  survey, 
clearly  was  nonrandom.  Invalid  addresses  were  most  likely  to  cut  out 
nonwhites,  the  less  educated,  and  the  less  intellectually  endowed  men. 
Overall  response  rates  also  were  lower  for  nonwhites,  the  less  educated, 
lower  AFQT  scorers,  single  men,  and  participants  in  Project  100,000. 

The  underrepresented  subgroups  within  the  population  undoubtedly  differ 
from  the  respondents  in  their  post-service  histories.  Note  that  this 
analysis  views  noncontacts  (invalid  addresses)  as  one  component  of  non¬ 
response  bias,  and  therefore  return  rates  are  somewhat  lower  than  they 
would  be  if  the  computations  omitted  invalid  addresses.  But  this  pro¬ 
cedure  is  entirely  consistent  with  the  idea  that  nonresponse  bias 
accrues  from  any  nonrandom  loss  of  respondents ,  whether  through  their 
refusal  to  participate  or  through  our  inability  to  contact  them. 


mi*****, 
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Table  3 


PERCENT  AND  NUMBER  OF  SEVERAL  CATEGORIES  OF 
ENLISTED  MEN  RESPONDING  TO  QUESTIONNAIRE 


Service 


AFQT 

Categories 


Marital 

Status 


Entrance 

Status 


Education 


Army 

Air  Force 
Navv 


Caucasian 

Negro 


Class  I 
Class  II 
Class  III 
Class  IV 
Class  V 


Married 

Single 

Unknown3 


Enlisted 

Inducted  (all  Army) 


Less  than  8  vr 

8  yr 

9  yr 

10  yr 

11  yr 

12  vr 

1  yr  college 

2  yr  college 

3  yr  college 
College  graduate 
M.  A. 

Ph.D. 


94, 

,032 

131, 

,  108 

6, 

,998 

9, 

,387 

8, 

,843 

1*171 

109, 

,8  75 

1  3, 

,  666 

100, 

,181 

144, 

,5  34 

,694 

1», 

,132 

109, 

,875 

16  3, 

,  666 

8, 

,602 

12, 

,845 

37, 

,139 

53, 

,  380 

40, 

,596 

58, 

,778 

22 , 

,871 

37, 

,708 

577 

955 

109^ 

,875 

iST 

,  bb<-. 

50, 

,310 

63 , 

,  799 

50, 

i-T» 

O' 

00 

88, 

,833 

8, 

,997 

9,014 

109, 

,875 

163, 

,666 

41, 

,686 

72 

,354 

68, 

,  189 

Q1 

,312 

109, 

,875 

16  3 

,666 

800 

1 

,203 

o 

,208 

3 

.531 

3, 

,225 

5 

,2  30 

4 

,339 

7 

.524 

5 

,022 

8 

,624 

66, 

,796 

99 

,960 

10, 

,138 

14 

,873 

7 

,265 

10 

,053 

3, 

,347 

4 

,506 

6 

,183 

7 

.50  3 

268 

329 

285 

328 

IbT 

,875 

1M 

,666 

not 

ind i cat 

e  maritt 

ll 

In  contrast. 

the  nai 

rita 

ssumed  to  be 

the  same  a' 
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Clearly  the  post-service  mail  questionnaire  data  contain  a  generous  dose 
of  nonresponse  bias.  The  issue  is,  what  can  be  done  about  it? 

SOME  POSSIBLE  REMEDIES  FOR  NONRESPONSE  BIAS 

The  worse  a  mouthwash  tastes,  the  more  we  conclude  it  works.  The 
more  we  pay  for  something,  the  less  dispassionately  we  can  see  its  flaws, 
and  the  more  severe  the  initiation  required  to  gain  membership  in  a 
group,  the  more  we  value  the  group.  These  are  all  examples  of  the  dis¬ 
sonance  theory  proposition  that  we  cherish  things  which  require  great 
* 

effort  to  attain.  The  more  effort  we  expend,  the  more  we  seek  to 
justify  our  efforts. 

Researchers  who  use  mail  questionnaires  are  no  strangers  to  cog¬ 
nitive  dissonance.  Once  a  great  deal  of  time,  energy  and  money  have 
gone  into  a  mail  survey,  it  is  difficult  to  admit  that  nonresponse 
bias  may  have  made  the  effort  all  for  naught.  Faced  with  a  low  response 
rate  and  evidence  of  nonresponse  bias,  the  researcher  must  make  some 
painful  decisions. 

One  option  is  to  give  up  any  hope  of  estimating  population  parame¬ 
ters.  In  this  case,  estimates  are  computed  for  those  sul populations 
which  showed  an  adequate  response  rate,  and  no  attempt  is  made  to  aggre¬ 
gate  these  subpopulation  estimates  into  sample  estimates  of  population 
parameters.  This  may  be  a  technically  safe  way  to  handle  the  problem 
of  nonresponse  bias,  but  it  certainly  is  not  very  appealing  to  the  re¬ 
searcher  who  has  spent  a  great  deal  of  time  and  effort  trying  to  esti¬ 
mate  population  parameters;  and  it  may  be  an  unnecessarily  severe  waste 
of  data.  It  is  very  likely  that  the  survey  can  be  "patched  up." 

One  often  suggested  way  of  patching  up  a  mail  survey  is  to  replace 
missing  respondents  either  through  direct  substitution  or  oversampling. 
If,  for  example,  1,000  questionnaires  are  mailed  out,  and  only  500  are 
returned,  we  can  still  get  a  total  of  1,000  completed  questionnaires 
simply  by  sending  questionnaires  to  another  1,000  people  and  trusting 
that  their  response  rate  will  also  be  something  like  50  percent.  This 


There  is  a  substantial  amount  of  laboratory  and  field  research 
supporting  aspects  of  this  theory,  but  it  is  not  without  flaws;  and 
the  simple  effort  justification  hypothesis  requires  qualification.  See, 
for  example,  Festinger  (1957),  Aronson  and  Mills  (1959),  Gerard  and 
Mathewson  (a966),  and  Doob,  et  al.  (1969). 
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illustrates  the  ex  •post  facto  replacement  of  missing  respondents.  If 
prior  experience  has  shown  that  a  particular  subgroup  has  a  low  response 
rate,  we  can  oversample  at  the  outset.  That  is,  we  can  send  out  more 
questionnaires  to  people  in  the  low  response  subgroup.  For  example, 
say  we  have  two  equal  size  subgroups,  A  and  B ,  within  a  population; 
subgroup  A  usually  has  a  100  percent  return  rate,  but  subgroup  B  typi¬ 
cally  shows  only  a  SO  percent  response  rate.  To  make  the  final  returns 
reflect  the  population  split,  we  can  oversample  members  of  subgroup  B\ 
to  be  precise,  we  can  sample  twice  as  many  members  of  subgroup  B  as  of 
subgroup  A. 

Replacement  strategies  have  the  effect  of  narrowing  the  sample 
estimate  of  the  population  variance,  and  thus  the  precision  of  survey 
estimates  is  improved,  but  this  does  not  reduce  the  nonresponse  bias 
which  impacts  upon  the  overall  accuracy  of  survey  estimates.  It  is 
important  that  we  not  confuse  the  problem  of  sample  size  reduction  due 
to  nonresponse  with  the  more  central  problem  of  bias  due  to  nonresponse. 
(Cf.  Kish,  1965,  p.  549) .  Oversampling  merely  replaces  missing  respon¬ 
dents  with  'ople  who  are  just  like  the  ones  who  responded  in  the  first 
place!  We  still  have  a  biased  sample;  it  is  simply  larger,  and  this 
ignores  the  heart  of  the  matter.  The  problem  is  not  the  size  of  the 
sample;  it  '■  the  composition  of  the  sample. 

Weighting  returns  is  another  strategy  for  handling  nonresponse 
bias;  but,  alas,  it  relies  upon  the  same  assumptions  as  the  replacement 
strategy.  It  is,  in  effect,  a  kind  of  pretend  replacement  strategy, 
however,  when  the  decision  is  made  to  patch  up  the  survey  data,  weighting 
is  usually  preferred  to  other  strategies  even  though  we  would  prefer  to 
have  the  larger  sample  size  that  replacement  strategies  produce. 

WILL  WEIGHTING  SOLVE  THE  PROBLEM  OF  NONRESPONSE  BIAS? 

Weighting  is  usually  preferred  to  replacement  methods  because 
weighting  (a)  costs  less,  (b)  can  be  implemented  after  the  survey  is 
completed,  and  (c)  eliminates  many  of  the  problems  of  field  substitution. 
And,  although  weighting  is  not  guaranteed  to  eliminate  bias,  it  will, 
under  some  circumstances,  attenuate  the  bias  in  estimates  of  central 
tendency  and  variance. 
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The  Logic  of  Weighting 

Weighting  schemes  amount  to  counting  some  cases  more  than  others 
in  order  to  compensate  for  the  underrepresentation  of  certain  subgroups. 
Consider  this  simple  illustration:  Mail  questionnaires  are  sent  to  100 
Cadillac  owners  and  100  Volkswagon  owners.  All  of  the  Cadillac  owners 
respond,  but  only  25  of  the  VW  owners  reply.  The  returns  clearly 
underrepresent  small  car  drivers,  so  we  weight  the  VW  drivers'  data  by 
the  reciprocal  of  the  response  rate  for  this  subgroup,  that  is,  4.  In 
effect,  every  VW  owner  who  responds  gets  four  "votes"  in  figuring  popu¬ 
lation  estimates,  whereas  Cadillac  drivers  get  only  one  vote  apiece. 

Weighting  gives  an  arithmetically  pleasing  solution  to  the  problem 
of  nonresponse,  but  it  can  sometimes  produce  disastrous  results  because 
all  weighting  schemes  assume  that  (a)  the  respondents  represent  a  random 
sample  of  their  subgroup  and  (b)  we  have  taken  all  the  relevant  differ¬ 
entia  between  respondents  into  consideration  in  defining  subgroups. 

Both  assumptions  are  tenuous. 

First,  the  reason  for  weighting  s  to  eliminate  the  nonresponse  bias 
which  has  resulted  from  the  nonrandom  attrition  (self-selection)  of 
respondents.  By  definition,  the  respondents  are  not  a  random  sample 
of  all  potential  respondents;  if  chey  were,  we  would  not  have  a  non¬ 
response  bias  problem  in  the  first  place  and  there  would  be  no  need 
for  adjustment!  For  example,  in  the  survey  of  car  owners,  it  may  be 
that  the  VW  owners  who  responded  use  their  small  cars  as  a  second  car; 
they  are  probably  wealthier,  more  educated,  and  older  than  VW  owners 
who  use  their  cars  as  basic  transportation.  Thus  the  weighting  scheme 
would  overestimate  the  average  wealth,  education,  and  age  of  VW  owners 
relative  to  Cadillac  owners. 

Second,  we  can  never  identify  all  of  the  dimensions  that  are 
"relevant"  in  weighting;  and  as  in  stratified  sampling,  it  is  rarely 
possible  to  consider  more  than  two  or  three  dimensions.  The  problem 
of  weighting  on  all  relevant  dimensions  is  not  unlike  the  problem  of 
matching  to  achieve  group  equivalence;  matching  is  never  a  substitute 
for  randomization  because  we  can  never  be  sure  that  we  have  taken  all 
of  the  relevant  variables  into  consideration.  (cf.  Campbell  and  Stanley, 
1966). 
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On  a  less  theoretical  plane,  there  are  also  some  practical  con¬ 
straints  to  weighting.  For  one,  we  rarely  have  enough  data  about  all 
potential  respondents  to  make  weighting  possible.  The  efficacy  of 
weighting  is  detetrm<r>pj  by  the  amount  of  variance  in  the  dependant 
variable  that  the  weighting  variables  can  collectively  explain.  For 
example,  age  and  education  may  account  for  ninety  percent  of  the  vari¬ 
ance  in  the  income  of  VW  owners,  but  will  these  variables  explain  most 
of  the  variance  in  other  dependent  variables  such  as  number  of  miles 
driven  per  year?  It  may  be  necessary  to  weight  the  returns  differently 
for  different  dependent  variables. 

In  sum,  weighting  strategies  rest  on  tenuous  and  unverifiable 

assumptions,  and  while  weighting  on  multiple  dimensions  can  reduce  the 

magnitude  of  nonresponse  bias,  it  is  probably  safe  to  say  that  weighting 

schemes  can  rarely  eliminate  nonresponse  bias.  But  situations  sometimes 

arise  in  which  any  estimate,  even  a  very  biased  one,  is  better  than  no 

information  at  all,  and  when  this  is  the  case  and  some  data  are  available 

* 

for  all  potential  respondents,  weighting  may  be  a  tolerable  strategy 
for  attenuating,  but  not  eliminating,  nonresponse  bias.  This  is  illus¬ 
trated  in  the  next  section  where  we  return  to  the  post-service  survey 
data  to  demonstrate  the  mechanics  of  weighting. 

The  Mechanics  of  Weighting 

Weighting  involves  three  steps.  First,  the  best  set  of  weighting 
variables  must  be  identified.  This  can  be  done  through  regression 
analyses  in  which  linearity  is  assumed  and  response/no  response  is 
taken  as  a  dummy  dependent  variable.  Second,  weights  are  created  by 
figuring  the  reciprocal  of  the  response  rate  for  all  of  the  subgroups 
which  are  defined  by  cross-tabulating  the  weighting  variables.  And, 
finally,  the  weights  are  applied  to  the  distribution  of  raw  scores  in 
order  to  produce  adjusted  estimates  of  population  parameters. 


We  are  probably  much  too  quick  to  assign  problems  to  this  cate¬ 
gory  when  in  fact  incorrect  information  may  be  more  harmful  than  no 
information  at  all. 
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In  most  surveys  we  have  little  or  no  information  about  all  of  the 
respondents,  and  therefore  the  choice  of  weighting  variables  is  greatly 
constrained.  But  in  some  cases,  and  the  post-service  survey  discussed 
earlier  is  one  of  these  rare  cases,  we  have  extensive  information  about 
respondents  and  nonrespondents  alike;  and  this  information  can  be  used 
to  create  subgroups  for  weighting  purposes.  The  issue  then  becomes 
one  of  finding  that  set  of  variables  which  best  accounts  for  the  vari¬ 
ance  in  response/nonresponse.  In  the  post-service  data,  for  example, 
three  variables  account  for  almost  87  percent  of  the  variance.  These 
variables  are  Service,  Race,  and  AFQT  score,  and  they  would  seem  to  be 
good  candidates  for  weighting  dimensions  in  the  post-service  survey. 

To  compute  the  weights,  we  assign  all  potential  respondents  to 
mutually  exclusive  and  exhaustive  subgroups  according  to  their  value  on 
the  weighting  variables,  and  then  compute  the  response  rate  (proportion) 
for  each  subgroup.  The  weight  for  each  subgroup  is  simply  the  recipro¬ 
cal  of  its  response  rate.  To  grasp  the  mechanics  of  this  procedure,  see 
Tables  4  and  5.  Table  4  shows  what  proportion  in  each  subgroup  responded 
to  the  post-service  mail  questionnaire,  and,  of  course,  the  cells  (sub¬ 
groups)  are  defined  by  the  three  weighting  variables:  Service,  Race, 
and  AFQT  class.  Table  5  shows  the  same  three-way  cro3s-tabulation,  but 
here  the  cell  entries  are  the  reciprocals  of  those  in  Table  4.  These 
are  the  weights  applied  to  the  raw  data  to  obtain,  for  example,  weighted 
averages  of  post-service  civilian  earnings  or  weighted  averages  of  other 
relevant  dependent  variables. 

The  pattern  of  nonresponse  which  occurred  in  the  post-service 
survey  of  enlisted  men  was  not  random.  Invalid  addresses  were  related 
to  service  affiliation,  AFQT  class,  ethnicity,  education,  and  enroll¬ 
ment  in  special  training  programs  (e.g..  Project  100,000).  Non¬ 
returns  were  also  nonrandom;  response  was  relat'd  to  service  affiliation, 
AFQT  class,  ethnicity,  education,  mar^al  status  and  participation  in 
special  programs.  Taken  together,  the  noncontacts  and  nonreturns  led 
to  the  underrepresentation  of  nonwhites,  the  less-educated  and  the  less 
intellectually  endowed  men.  But  the  pattern  of  nonresponse  in  this 
particular  survey  had  little  impact  on  our  estimates  of  the  population 
central  tendency  or  variance  for  any  of  the  survey  variates,  because 
the  subpopulation  which  produced  the  low  return  rate  (i.e.,  the  Navy) 
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Table  4 


PROPORTION  OF  EACH  SUBGROUP  WHO  RESPONDED  WHERE 
SUBGROUPS  ARE  DEFINED  BY  SERVICE,  RACE,  AND  AFQT  CLASS 


Service 

AFQT 

Caucasian 

N 

Negro 

N 

Claaa 

I 

.819 

7314 

.615 

26 

Claaa 

II 

.781 

37437 

.587 

mm 

Army 

Class 

III 

.738 

46171 

.523 

EmI 

Class 

IV 

.690 

21996 

.515 

11966 

Class 

V 

.679 

470 

.534 

483 

Total 

.748 

113388 

.520 

17720 

Class 

I 

.817 

984 

.750 

4 

Class 

II 

.779 

4103 

.590 

61 

Air  Force 

Class 

III 

.736 

3077 

.422 

249 

Class 

IV 

.719 

664 

.482 

243 

Class 

V 

.000 

2 

**** 

A  A** 

Total 

.763 

8830 

.469 

557 

Class 

I 

.417 

4493 

.250 

24 

Class 

II 

.388 

11108 

.221 

131 

Navy 

Class 

III 

.369 

4413 

.281 

160 

Class 

IV 

.353 

2302 

.261 

540 

Class 

V 

**** 

**** 

**** 

**** 

Total 

.387 

22316 

.259 

855 
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Table  5 

SUBGROUP  WEIGHTS  (REFER  TO  TABLE  4) 


Service 

AFQT 

Caucasian 

Negro 

Claas  I 

1.22 

1.63 

Class  II 

1.28 

1.70 

A TWJ 

Class  III 

1.36 

1.91 

Class  IV 

1.45 

1.94 

Class  V 

1.47 

1.87 

Class  I 

1.22 

1.33 

Class  II 

1.29 

1.69 

Air  Force 

Class  III 

1.36 

2.37 

Class  IV 

1.40 

2.07 

Class  V 

**** 

**** 

Class  I 

2.40 

4.00 

Class  II 

2.58 

4.52 

Navy 

Class  III 

2.70 

3.56 

Class  IV 

2.84 

3.83 

Class  V 

**** 

**** 

5 ,  JVf'  «  J>« 
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conscituced  a  relatively  small  portion  of  the  overall  population  (about 
14.2  percent,  to  be  exact).  Nonresponse  would  have  posed  a  problem  in 
this  survey  only  if  (a)  the  high  and  low  response  subpopulations  had 
been  about  equal  in  size,  and  (b)  nonresponse  was  correlated  with  the 
dependent  variables  of  interest.  Nonresponse  bias  occurs  when  the 
pattern  of  nonresponse  is  correlated  with  the  variables  which  the  survey 
is  designed  to  measure,  and  the  problem  is  that  we  have  no  way  of 
knowing  when  this  is  the  case. 
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